1. Field of the Invention
The present invention relates to an apparatus, method and program for processing an image and, in particular, to an image processing technique for obtaining a high-quality image taking into consideration a motion image blur on the image.
2. Description of the Related Art
When a moving image taken at a high-speed shutter or an animation is displayed on a display device such as a projector or a display, the motion of a moving object contained in an image can be displayed in a discontinued fashion. This leads to frequent image degradation in which a user sees multiple images of the moving object. The degradation of the moving image due to motion unnaturalness is generally referred to as motion jerkiness. On the other hand, when a moving image taken at a low-speed shutter such as with an open shutter is displayed, the image of an object may lack detail or an edge of the image becomes blurred because of the effect of motion blur. This phenomenon is referred to as motion blur, which is also one of the image degradations.
The principle of the generation of jerkiness and blur is described with reference to FIGS. 26-28. In accordance with vision characteristics, humans are known to visually recognize a light ray incident on the eyes as a value that results from integrating the light ray for a predetermined period of time. FIGS. 26-28 diagrammatically illustrate the way an object is viewed by a viewer in accordance with the vision characteristics.
FIGS. 26A and 26B illustrate how a still object and a moving object look in the real world.
FIG. 26A illustrate chronological movement of a still object 71 and a moving object 72 with the abscissa representing position (x) and the ordinate representing time (t). FIG. 26B diagrammatically illustrates the vision of a viewer who views the still object 71 and the moving object 72. The viewers views the objects in two vision conditions, i.e., a tracking vision tracking the moving object 72 and a fixed vision not tracking the moving object 72, respectively illustrated as (a) tracking vision and (b) fixed vision.
If the moving object 72 is tracked by the viewer in (a) tracking vision as illustrated in FIG. 26B, the moving object 72 looks like moving object vision information a72. This is identical to fixed object vision information b71 in which the still object 71 looks in (b) fixed vision in FIG. 26B. If the viewer views the moving object 72 in the tracking vision in this way, the moving object 72 looks in the same way as the still object 71 looks in the fixed vision. If the viewer views the moving object 72 in the fixed vision as in (b) fixed vision illustrated in FIG. 26B, the moving object 72 looks like moving object vision information b72 in FIG. 26B. The viewer visually recognizes the moving object as a continuously moving object, and is free from any discomfort viewing.
FIGS. 27A and 27B illustrate the principle of the generation of jerkiness viewed by the viewer when a moving image taken at a high-speed shutter or an animation is displayed on a display device such as a projector or a display device. Jerkiness is a phenomenon in which the motion of a moving object contained in an image is displayed in a discontinuous manner, causing the viewer to visually recognize multiple images of the moving object.
A moving object in the real world in FIGS. 26A and 26B is imaged at a high-speed shutter, and then displayed on a display device at a refresh rate of 60 Hz. FIGS. 27A and 27B illustrate the way the viewer visually recognizes the moving object. FIG. 27A illustrates a change in the display positions of a display still object 81 and a display moving object 82 on a display device. In FIG. 27A, the ordinate represents time (t) and is graduated in refresh periods of the display device (each period being 1/60 second), and the abscissa represents display position (x). FIG. 27B diagrammatically illustrates the vision status of the viewer who views the display still object 81 and the display moving object 82 displayed on the display device. The vision status of the viewer includes (a) tracking vision in which the viewer views the image with the display moving object 82 being tracked, and (b) fixed vision in which the viewer views the image with the display moving object 82 not tracked but with the vision of the viewer fixed.
When the display moving object 82 displayed on the display device is viewed by the viewer in (a) tracking vision as illustrated in FIG. 27B, an image a82 looks in the same way as the image a72 looks in (a) tracking vision in FIG. 26B. The viewer visually recognizes the image in the same way as the viewer views a still object in the fixed vision.
When the display moving object 82 displayed on the display device is viewed by the viewer in (b) fixed vision as illustrated in FIG. 27B, the display moving object 82 looks like images b82 not continuously moving but discontinuously moving in a manner different from the real world in the viewer's vision. As a result, the viewer visually recognizes as multiple images the moving object displayed on the display device based on the human vision characteristics. In human vision characteristics, humans visually recognize a light ray incident on the eyes as a value that results from integrating the light ray for a predetermined period of time.
The viewer thus views a single object as multiple objects. This phenomenon is called jerkiness. In principle, an object moving at a high speed suffers more from jerkiness. The lower the frame rate of the display device, the more jerkiness takes place, and the higher the frame rate, the less jerkiness takes place. Furthermore, jerkiness takes place more in a portion of an image where a change in spatial luminance is large, i.e., where a spatial contrast is high.
FIGS. 28A and 28B illustrate how a blur viewed by the viewer is generated when a moving image taken at a low-speed shutter such as with an open shutter or an animation is displayed on a display device such as a projector or a display. The blur is a phenomenon in which the image of an object may lack detail or an edge of the image becomes blurred because of the effect of motion blur.
FIGS. 28A and 28B diagrammatically illustrate how the moving object in the real world illustrated in FIGS. 26A and 26B looks to the viewer when the moving object is imaged at a low-speed shutter and then displayed on the display device at a refresh rate of 60 Hz. FIG. 28A illustrates a change in the display positions of a display still object 91 and a display moving object 92 on the display device. In FIG. 28A, the ordinate represents time (t) and is graduated in refresh periods of the display device (each period being 1/60 second), and the abscissa represents display position (x). FIG. 28B diagrammatically illustrates the vision status of the viewer who views the display still object 91 and the display moving object 92 displayed on the display device. The vision status of the viewer includes (a) tracking vision in which the viewer views the image with the display moving object 92 being tracked, and (b) fixed vision in which the viewer views the image with the display moving object 92 not tracked but with the vision of the viewer fixed.
When the display moving object 92 displayed on the display device is viewed by the viewer in (b) fixed vision as illustrated in FIG. 28B, an image b92 looks in the same way as the image b72 looks in (b) fixed vision in FIG. 26B. The viewer visually recognizes the moving object as a continuously moving object, and is free from any discomfort viewing.
When the display moving object 92 displayed on the display device is viewed by the viewer in tracking vision, an image a92 looks to the viewer as a blurred image as illustrated in FIG. 28B in a manner different from the case in which the still object is viewed in fixed vision. When the display moving object 92 in FIG. 28A is imaged, the motion of the moving object during a long exposure of a low-speed shutter is recorded within one frame, and the moving object is thus displayed as a band in one frame. Such a phenomenon is referred to as blur.
In principle, there is a trade-off between jerkiness and blur with respect to the imaging shutter speed. Mere shutter control causes either the degradation of jerkiness or the degradation of blur to be pronounced. More specifically, if an image taken at a relatively high shutter speed with respect to the frame rate of the moving image is displayed as a still image, high sharpness is provided. If the image is displayed as a moving image, the motion of a moving area within the image, in particular, a moving area at a high speed is not smooth. Such an image looks unnatural to the vision of humans. If an image taken at a relatively low shutter speed with respect to the frame rate of the moving image is displayed as a moving image, high sharpness is provided. If the image is displayed as a moving image, the motion of a high-speed moving area within the image is smooth, but the entire image lacks sharpness.
Japanese Unexamined Patent Application Publication No. 2007-274299 (WO07/114220), assigned to the same assignee of the present invention, discloses a jerkiness reducing technique intended to be used on an image taken at a high shutter speed. In accordance with the disclosed technique, a motion blur is added through image processing. An added amount of motion blur is controlled through analysis of image processing so that the excessive addition of the motion blur does not cause blur degradation. A technical approach of reducing the motion blur through image processing performed mainly on an input image taken at a low shutter speed is widely studied. For example, image processing techniques for correcting blur of an image include mainly an inverse convolution technique based on a blur model, and a technique not using no blur model, such as a peaking technique or a shock filter technique. For example, the technique disclosed in the paper entitled “Extension of Coupled Nonlinear Diffusion to Motion De-blurring—Introduction of Anisotropic Peaking,” Takahiro SAITO, Hiroyuki HARADA, and Takashi KOMATSU, The Institute of Image Information and Television Engineers Vol. 58, No. 12 pp. 1839-1844 (2004) is related to the blur-model-based inverse convolution technique as motion blur reduction means. The technique disclosed in the paper entitled “Motion De-blurring Using a Blur Model,” Takahiro SAITO, Hiroyuki HARADA, Taishi SANO, and Takashi KOMATSU, The Institute of Image Information and Television Engineers Vol. 59, No. 11, pp. 1714-1721 (2005) is related to the technique not using no blur model as motion blur reduction means.